Activation-Based Recursive Self-Organizing Maps A General Formulation and Empirical Results

We generalize a class of neural network models that extend the Kohonen self-organizing map (SOM) algorithm into the sequential and temporal domain using recurrent connections. Behaviour of the class of Activation-based Recursive Self-organizing Maps (ARSOM) is discussed with respect to the choice of transfer function and parameter settings. By comparing performances to existing benchmarks we demonstrate the robustness and systematicity of the ARSOM models, thus opening the door to practical applications. Introduction In this paper we offer a general definition of a class of models, which we call Activation-based Recursive Self-Organizing Maps (ARSOM), that extend the self-organizing map (SOM) algorithm [14, 15] to the sequential or temporal domain by adding recurrent connections to the map (or network) units. The SOM has been established as one of the most widely used techniques for unsupervised data clustering, i.e., in the absence of any a priori given classification. However, the SOM, as well as other popular artificial neural network architectures such as the back-propagation algorithm [19, 20] for supervised learning of data, was designed for data in which sequential order is insignificant, i.e., where the input elements are context-independent. The applicability of the fundamentally atemporal SOM to tasks such as decision-making and prediction in the context of preceding events is therefore limited. A number of extensions to these basic architectures have been proposed to overcome this may be referred to for a more complete view. However, within these attempts, an important distinction can be made between the representation of time explicitly, giving time essentially a spatial representation, versus representing time implicitly, " by the effect it has on processing " [5]. Explicit representations of time, e.g., those implemented via a time window, treat time statically as another vector dimension of the input. In contrast, implicit representations of time, e.g., those implemented through recurrent connections which feed back the network's own activation to itself, give the network a memory of previous events which is, in principle, capable of dynamically influencing subsequent processing. Thus with implicit representations of time the emphasis is on memory as a procedure rather than memory as a data structure. The basic approach of using recurrent connections to model temporal effects was first implemented within the back-3 propagation paradigm by Jordan [7] and Elman [5], and was applied to the SOM by Kaipainen et al. in the MuSeq model [9] and its extensions [10, 11]. The model of Voegtlin [25, 26] …